Skip to content

[kafka] Serve ApiVersions from registered capabilities - #4260

Open
gyang94 wants to merge 5 commits into
apache:mainfrom
gyang94:codex/kafka-api-versions
Open

gyang94 wants to merge 5 commits into
apache:mainfrom
gyang94:codex/kafka-api-versions

Conversation

@gyang94

@gyang94 gyang94 commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Closes #4265.

Part of umbrella #4185; this PR does not close the umbrella.

Implement ApiVersions negotiation using the capabilities actually registered on the server, and connect the Kafka request entry point to the asynchronous dispatcher. Depends on #4264 and its implementation PR #4259.

Brief change log

  • Add ApiVersionsHandler and build advertised API keys and version ranges from the capability registry.
  • Replace the request-handler switch and empty stubs with the asynchronous dispatcher.
  • Register only ApiVersions v0-v4 at this stage; Metadata is added in [kafka] Implement Metadata API over Fluss table metadata #4261.
  • Test future-version requests through the decoder, v0 error response encoding, correlation IDs, unknown tagged fields, buffer release, and subsequent negotiation on the same connection.
  • Verify advertised ranges, ordering, and hidden API filtering across ApiVersions v0-v4; cover invalid client software names and versions in v3 and v4.

Incremental scope relative to PR01: 5 files, +394 / -258 lines, including tests.

Tests

Validated with Java 11 in this worktree, using reactor dependencies built from the same source tree:

  • mvn -o -pl fluss-kafka -am clean verify '-Dtest=Kafka*Test,ApiVersionsHandlerTest,Kafka*ITCase' -Dsurefire.failIfNoSpecifiedTests=false: 37 unit tests and 2 integration tests passed, with zero failures, errors, or skips. Both Surefire lifecycle executions ran the selected set of 39 distinct cases.
  • Reactor Checkstyle, Spotless, and RAT checks passed; git diff --check passed.
  • git range-diff confirms that the original ApiVersions patch was preserved when rebasing onto PR01 274c00d7af9f0347adeb9372e5f67b41222eed2e; the additional commit strengthens regression coverage.

The entire repository test suite and the full producer-client version matrix were not run.

API and Format

ApiVersions reports registered capabilities. No Fluss public API or storage-format change. Produce, Metadata, authentication, and topic administration are not implemented in this stage.

Documentation

The supported-capability boundary is documented in this PR and the handler Javadoc.

Stack and review

  1. [kafka] Add request dispatch and transport framework #4259 — request and transport framework.
  2. This PR — ApiVersions.
  3. [kafka] Define DDL table mapping for Kafka compatibility #4277 — DDL table mapping.
  4. [kafka] Implement Metadata API over Fluss table metadata #4261 — Metadata.

This PR remains a draft until its prerequisite lands. The direct base is PR01 274c00d7af9f0347adeb9372e5f67b41222eed2e; this PR's head is 546d8e5dae253dab39f02c64ac5b94e2e6f82f57.

Because the target is Apache main, GitHub's Files changed view includes the prerequisite. For this stage alone, review the incremental comparison. Rebase onto the updated main after the prerequisite is merged and before marking ready.

Generative AI disclosure: Codex assisted with implementation, testing, and commit organization. Human review is required before merge.

gyang94 and others added 5 commits October 14, 2026 13:45
Route requests through the dispatcher and advertise only implemented APIs.
Return version-aware errors for unsupported APIs and invalid requests.

Validated with mvn -o -pl fluss-kafka verify (23 unit tests and 1 IT).

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 313/313
AI-Contributed/UT: 137/137
Exercise future-version decoding, v0 error serialization, correlation IDs,
unknown tagged fields, buffer release, and renegotiation on one connection.
Verify registry-defined ranges, hidden APIs, ordering, and invalid client
software fields across the supported ApiVersions schemas.

Validated with Java 11 reactor clean verify: 37 unit tests and 2 integration
tests passed, with Checkstyle, Spotless, RAT, and git diff checks.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 238/238
Construct the registry once with all handlers and defer ApiVersions registry
access until request handling. Advertised capabilities and dispatch continue
to share the same immutable registry without a separate capability list.
Update coverage to verify lazy access and handler identity across versions.

Validated with Java 11 reactor clean verify: 44 unit and 2 integration cases
passed in both Maven test executions, including inherited error-mapping and
idle regressions. Kafka main and test sources also compile with --release 8.
Spotless, Checkstyle, RAT, and diff checks passed.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 28/28
AI-Contributed/UT: 33/33
Use the dispatcher-backed handler constructor in the fallback test added by
PR01. Preserve its unsupported-version, listener, response, and buffer checks.

Validated with Java 11 reactor clean verify: 39 unit and 2 integration cases
passed in both Maven test executions. Kafka main and test sources compile
with --release 8; Spotless, Checkstyle, RAT, and diff checks passed.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 0/0
AI-Contributed/UT: 3/3
Check the deferred registry before reading advertised capabilities so an
initialization-order failure reports an explicit diagnostic message.

Validated with Java 11 reactor tests: 16 ApiVersions and request-handler
cases passed, along with Spotless, Checkstyle, RAT, and diff checks.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 4/4
AI-Contributed/UT: 0/0
@gyang94
gyang94 force-pushed the codex/kafka-api-versions branch from 127e031 to 062047d Compare October 14, 2026 05:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[kafka] Implement registry-backed ApiVersions API

1 participant